掌握 NLP 从基础到LLMs

Mastering NLP from Foundations to LLMs

将基于规则的先进技术应用于LLMs并使用 Python 解决现实世界的业务问题

Apply advanced rule-based techniques to LLMs and solve real-world business problems using Python

利奥尔·加齐特

Lior Gazit

梅萨姆·加法里

Meysam Ghaffari

掌握 NLP 从基础到LLMs

Mastering NLP from Foundations to LLMs

版权所有 © 2024 Packt 出版

Copyright © 2024 Packt Publishing

版权所有。未经出版商事先书面许可，不得复制本书的任何部分、将其存储在检索系统中或以任何形式或任何方式传播，批评文章或评论中嵌入的简短引文除外。

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

本书的编写过程中已尽一切努力确保所提供信息的准确性。然而，本书中包含的信息在出售时不提供任何明示或暗示的保证。作者、Packt Publishing 或其经销商和分销商均不对因本书直接或间接造成或声称造成的任何损害承担责任。

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing 致力于通过适当使用大写字母来提供本书中提到的所有公司和产品的商标信息。然而，Packt Publishing 无法保证该信息的准确性。

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

集团产品经理：Ali Abidi

Group Product Manager: Ali Abidi

出版产品经理：Ali Abidi

Publishing Product Manager: Ali Abidi

图书项目经理：Hemangi Lotlikar

Book Project Manager: Hemangi Lotlikar

内容开发编辑：Priyanka Soam

Content Development Editor: Priyanka Soam

技术编辑：Rahul Limbachiya

Technical Editor: Rahul Limbachiya

文案编辑：Safis 编辑

Copy Editor: Safis Editing

校对：萨菲斯编辑

Proofreader: Safis Editing

索引器：Rekha Nair

Indexer: Rekha Nair

制作设计师：Gokul Raj ST

Production Designer: Gokul Raj S.T

高级 DevRel 营销协调员：Vinishka Kalra

Senior DevRel Marketing Coordinator: Vinishka Kalra

首次发布：2024 年 4 月

First published: April 2024

生产编号：1040424

Production reference: 1040424

由...出版

Published by

帕克特出版有限公司

Packt Publishing Ltd.

格罗夫纳之家

Grosvenor House

11 圣保罗广场

11 St Paul’s Square

伯明翰

Birmingham

B3 1RB，英国。

B3 1RB, UK.

国际标准书号：978-1-80461-918-6

ISBN: 978-1-80461-918-6

www.packtpub.com

www.packtpub.com

致我的父母、我的兄弟姐妹和我的孩子们。

To my parents, my siblings, and my boys.

最重要的是，感谢我的真主，他的支持和信仰为本书的创作铺平了道路。

Above all, to my Alla, whose support and faith paved the path to the creation of this book.

谢谢你！

Thank you!

——利奥尔

– Lior

前言

Foreword

自然语言处理( NLP ) 是一个令人困惑的问题的核心：人类和计算机这两个截然不同的实体如何才能真正相互沟通？人类语言是社会和生物进化的复杂、不完美的产物。它充满了不合逻辑的例外、微妙的细微差别和多层次的抽象思维。相比之下，计算机通过数学模型进行通信，无论多么复杂，数学模型都遵循一组逻辑的、可验证的规则。随着数字系统在人类活动中发挥越来越大的作用，它们必须能够正确解释人类所说的话的实际含义。

Natural language processing (NLP) lies at the heart of a perplexing question – how can two radically different entities – humans and computers – truly communicate with one another? Human language is the complex, imperfect product of social and biological evolution. It’s filled with illogical exceptions, subtle nuance, and multiple levels of abstract thinking. In contrast, computers communicate via mathematical models that, however complex, follow a logical, verifiable set of rules. As digital systems assume an ever greater role in human activity, they must be able to correctly interpret what humans actually mean from the words they say.

Lior Gazit 和 Meysam Ghaffari 的新书《Mastering NLP from Foundations to LLMs》是实现这一目标的重要资源。这本书是为从事文本工作的技术专业人士（从初学者到经验丰富的 NLP 专业人士）编写的，为应对本世纪最艰巨的挑战之一制定了实用策略。它通过 NLP 和大型语言模型( LLM )的复杂领域绘制了细致的课程，引导您了解基本概念，直至当代人工智能的顶峰。

Lior Gazit and Meysam Ghaffari’s new book, Mastering NLP from Foundations to LLMs, is a monumental resource for making that happen. Written for technology professionals who work with text – from beginners to seasoned NLP pros – the book lays out a practical strategy for one of this century’s most daunting challenges. It charts a meticulous course through the intricate realms of NLP and large language models (LLMs), guiding you through fundamental concepts to the apex of contemporary artificial intelligence.

这本书借鉴了 Gazit 对快节奏金融世界的沉浸感和 Ghaffari 在医疗保健领域的创新 NLP 发展。结果是技术深度和实际相关性之间实现了罕见的平衡。他们综合的专业知识塑造了这本书，内容丰富，实用见解丰富。每位作者的独特影响力丰富了故事内容——结合了 Gazit 对机器学习( ML ) 推动增长力量的理解和 Ghaffari 应用 ML造福社会的人文方法。

The book draws from Gazit’s immersion in the fast-paced world of finance and Ghaffari’s innovative NLP development in the healthcare sector. The result is a rare balance between technical depth and practical relevance. Their combined expertise shapes a book as rich in information as it is robust in practical insights. Each author’s distinct influence enriches the narrative – combining Gazit’s understanding of the power of machine learning (ML) to drive growth and Ghaffari’s humanistic approach to applying ML for societal good.

Gazit 和 Ghaffari 在支持 NLP 复杂算法的基本数学和统计支柱方面奠定了坚实的基础。他们采用从基本原理到高级应用逐步构建的教学策略，确保清晰的学习和理解轨迹。

Gazit and Ghaffari lay a solid foundation in the essential mathematical and statistical pillars supporting the complex algorithms of NLP. They employ a pedagogical strategy that progressively builds from basic principles to advanced applications, ensuring a clear trajectory of learning and comprehension.

随着叙述的进展，它深入探讨了机器学习模型的工程。我们将指导您构建模型、应用以及拟合性与普遍性之间的微妙平衡。对文本预处理的探索是彻底的，为您提供了有效准备 NLP 任务数据的基本工具，从标记化到命名实体识别的微妙艺术。

As the narrative progresses, it delves deep into the engineering of ML models. You are guided through model construction, application, and the nuanced balance of fit versus generalizability. The exploration of text preprocessing is thorough, equipping you with essential tools to effectively prepare data for NLP tasks, from tokenization to the subtle art of named entity recognition.

本书的核心内容——LLMs——以细心和深厚的专业知识的融合而揭开面纱。作者阐明了标志着LLMs崛起的理论基础、发展挑战和突破，引导您思考这些强大技术的方向。作者还提供了有关建立和访问LLMs的实用建议，为您提供在工作中利用这些模型的可行路径。这揭开了将高级模型集成到实际用例中这一有时令人畏惧的任务的神秘面纱。

The book’s centerpiece – LLMs – is unveiled with a blend of care and deep expertise. The authors articulate the theoretical foundations, developmental challenges, and breakthroughs that mark the ascent of LLMs, guiding you to contemplate the direction of these formidable technologies. The authors also provide practical advice on setting up and accessing LLMs, providing you with actionable paths to harness these models in your work. This demystifies the sometimes daunting task of integrating advanced models into practical use cases.

在富有远见的章节中，本书深入探讨了 RAG 和 LangChain 等先进技术的功能，让人们一睹人工智能管理日益复杂的任务的自动化未来。这种叙述不仅具有教育意义，而且具有启发性，揭示了LLMs在提高绩效和促进更大创新方面的潜力。

In its visionary chapters, the book plunges into the functionalities of advanced technologies such as RAG and LangChain, providing a glimpse into an automated future where AI manages increasingly complex tasks. This narrative not only educates but inspires, mapping out the potential of LLMs to enhance performance and facilitate even greater innovation.

最后通过一系列专家访谈，从基础到LLMs掌握 NLP提供了多种视角，通过各个行业的实际应用丰富了叙述。这本非凡的书展示了 NLP 和LLMs推动的广泛变革——为技术专业人士提供了一个详细的路线图，让他们自己可以在未来发挥重要作用。对于任何想要获得LLMs席位的人来说，这是一本必读之书。

Concluding with a series of expert interviews, Mastering NLP from Foundations to LLMs offers a diverse array of perspectives that enrich the narrative with real-world applications across various industries. This remarkable book showcases the widespread transformation being driven by NLP and LLMs – giving technology professionals a detailed roadmap to a future in which they themselves can have a major role. It is a must-read for anyone who wants a seat at the LLM table.

阿莎·萨克塞纳,

Asha Saxena,

企业家、教授和人工智能战略家

Entrepreneur, Professor, and AI Strategist

简介：https://ashasaxena.com/about-bio/

Bio: https://ashasaxena.com/about-bio/

贡献者

Contributors

关于作者

About the authors

Lior Gazit是一位技术精湛的 ML 专业人士，在建立和领导使用 ML 推动业务增长的团队方面拥有良好的成功记录。他是 NLP 领域的专家，并成功开发了创新的 ML 管道和产品。他拥有硕士学位，并在同行评审的期刊和会议上发表过文章。作为金融领域机器学习团队的高级总监和新兴初创企业的首席机器学习顾问，Lior 是业内受人尊敬的领导者，拥有丰富的知识和经验可供分享。Lior 充满热情和灵感，致力于使用机器学习来推动其组织的积极变革和发展。

Lior Gazit is a highly skilled ML professional with a proven track record of success in building and leading teams that use ML to drive business growth. He is an expert in NLP and has successfully developed innovative ML pipelines and products. He holds a master’s degree and has published in peer-reviewed journals and conferences. As a senior director of a ML group in the financial sector and a principal ML advisor at an emerging start-up, Lior is a respected leader in the industry, with a wealth of knowledge and experience to share. With much passion and inspiration, Lior is dedicated to using ML to drive positive change and growth in his organizations.

Meysam Ghaffari是一位高级数据科学家，在 NLP 和深度学习方面拥有深厚的背景。他目前在 MSKCC 工作，专门开发和改进针对医疗保健问题的 ML 和 NLP 模型。他在 ML 领域拥有超过 9 年的经验，在 NLP 和深度学习领域拥有超过 4 年的经验。他在佛罗里达州立大学获得计算机科学博士学位，在伊斯法罕理工大学获得计算机科学人工智能硕士学位，在伊朗科技大学获得计算机科学学士学位。在加入 MSKCC之前，他还曾在威斯康星大学麦迪逊分校担任博士后研究员。

Meysam Ghaffari is a senior data scientist with a strong background in NLP and deep learning. He currently works at MSKCC, where he specializes in developing and improving ML and NLP models for healthcare problems. He has over nine years of experience in ML and over four years of experience in NLP and deep learning. He received his PhD in computer science from Florida State University, his MS in computer science – artificial intelligence from Isfahan University of Technology, and his BS in computer science from Iran University of Science and Technology. He also worked as a post-doctoral research associate at the University of Wisconsin-Madison before joining MSKCC.

关于审稿人

About the reviewers

Amreth Chandrasehar是云、AI/ML 工程、可观测性、自动化和 SRE 领域的工程领导者。在过去的几年里，Amreth 在各个组织的云迁移、生成式 AI、AIOps、可观察性和 ML 采用方面发挥了关键作用。Amreth 也是 Conduktor Platform 平台的共同创建者以及多家 Observability 公司的技术/客户咨询委员会成员。Amreth 还共同创建并开源了服务健康仪表板工具 Kardio.io。Amreth 曾受邀在多个重要会议上发表演讲，并获得多项奖项。我要感谢我的妻子 Ashwinya 和儿子 Athvik 在我审阅本书期间的耐心和支持。

Amreth Chandrasehar is an Engineering Leader in Cloud, AI/ML Engineering, Observability, Automation, and SRE. Over the last few years, Amreth has played a key role in Cloud Migration, Generative AI, AIOps, Observability, and ML adoption at various organizations. Amreth is also a co-creator of the Conduktor Platform Platform and a Tech/Customer Advisory board member at various companies on Observability. Amreth has also co-created and open-sourced Kardio.io, a service health dashboard tool. Amreth has been invited and spoken at several key conferences and has received several awards. I would like to thank my wife Ashwinya and my son Athvik for their patience and support during my review of this book.

Shivani Modi是一位数据科学家，在机器学习、深度学习和 NLP 方面拥有丰富的背景，并获得了哥伦比亚大学的硕士学位。她的职业生涯横跨 IBM、SAP、C3 AI 等职位，并在 Konko AI 担任领导职务，专注于可扩展的 AI 模型和创新的LLMs工具。Shivani 对指导和道德人工智能应用的奉献精神通过她的咨询角色和对技术社会效益的承诺显而易见。她即将开展的项目旨在提高开发人员对LLMs的利用率，优先考虑安全性和效率。

Shivani Modi is a data scientist with a rich background in machine learning, deep learning, and NLP, having earned her Master’s from Columbia University. Her career spans roles at IBM, SAP, C3 AI, and leadership at Konko AI, focusing on scalable AI models and innovative LLM tools. Shivani's dedication to mentoring and ethical AI application is evident through her advisory roles and commitment to the societal benefits of technology. Her upcoming projects aim to enhance LLM utilization for developers, prioritizing security and efficiency.

免责声明

Disclaimer

本书由作者、技术专家和专业出版团队共同创作。书中表达的观点仅代表作者而非其雇主。

This book has been created by authors, technical experts, and a professional publishing team. The views expressed in the book are of the authors and not their employers.

前言

Preface

本书深入介绍了自然语言处理( NLP ) 技术，从机器学习( ML )的数学基础开始，一直到高级 NLP 应用程序，例如大型语言模型( LLM ) 和人工智能应用程序。作为学习体验的一部分，您将掌握线性代数、优化、概率和统计，这对于理解和实现 ML 和 NLP 算法至关重要。您还将探索一般的 ML 技术并了解它们与 NLP 的关系。在学习如何执行文本分类之前，将介绍文本数据的预处理，包括清理和准备文本进行分析的方法，文本分类是根据文本内容为其分配标签或类别的任务。LLMs的理论、设计和应用的高级主题将在本书的最后讨论，自然语言处理的未来趋势也将讨论，其中将包含专家对该领域未来的意见。为了增强您的实践技能，您还将研究模拟的现实世界 NLP 业务问题和解决方案。

This book provides an in-depth introduction to natural language processing (NLP) techniques, starting with the mathematical foundations of machine learning (ML) and working up to advanced NLP applications such as large language models (LLMs) and AI applications. As part of your learning experience, you’ll get to grips with linear algebra, optimization, probability, and statistics, which are essential for understanding and implementing ML and NLP algorithms. You’ll also explore general ML techniques and find out how they relate to NLP. The preprocessing of text data, including methods for cleaning and preparing text for analysis, will follow, right before you learn how to perform text classification, which is the task of assigning a label or category to a piece of text based on its content. The advanced topics of LLMs’ theory, design, and applications will be discussed toward the end of the book, as will the future trends in NLP, which will feature expert opinions on the future of the field. To strengthen your practical skills, you’ll also work on mocked real-world NLP business problems and solutions.

这本书适合谁

Who this book is for

本书面向技术人员，包括深度学习和 ML 研究人员、NLP 实践者、ML/NLP 教育工作者以及 STEM 学生。在项目中使用文本的专业人士以及现有的 NLP 从业者也将在本书中找到大量有用的信息。初级 ML 知识和 Python 基本工作知识将帮助您充分利用本书。

This book is for technical folks, ranging from deep learning and ML researchers, hands-on NLP practitioners, and ML/NLP educators, to STEM students. Professionals working with text as part of their projects and existing NLP practitioners will also find plenty of useful information in this book. Beginner-level ML knowledge and a basic working knowledge of Python will help you get the best out of this book.

本书涵盖的内容

What this book covers

第 1 章，探索 NLP 领域：综合介绍，解释了本书的内容、我们将涵盖哪些主题以及谁可以使用本书。本章将帮助您决定这本书是否适合您。

Chapter 1, Navigating the NLP Landscape: A Comprehensive Introduction, explains what the book is about, which topics we will cover, and who can use this book. This chapter will help you decide whether this book is the right fit for you or not.

第 2 章，掌握机器学习和 NLP 的线性代数、概率和统计，分为三个部分。在第一部分中，我们将回顾本书不同部分所需的线性代数基础知识。在下一部分中，我们将回顾统计学的基础知识，最后，我们将介绍基本的统计估计器。

Chapter 2, Mastering Linear Algebra, Probability, and Statistics for Machine Learning and NLP, has three parts. In the first part, we will review the basics of linear algebra that are needed at different parts of the book. In the next part, we will review the basics of statistics, and finally, we will present basic statistical estimators.

第 3 章，释放 NLP 中的机器学习潜力，讨论 ML 中可用于解决 NLP 问题的不同概念和方法。我们将讨论一般特征选择和分类技术。我们将涵盖机器学习问题的一般方面，例如训练/测试/验证选择以及处理不平衡数据集。我们还将讨论评估 NLP 问题中使用的 ML 模型的性能指标。我们将解释这些方法背后的理论以及如何在代码中使用它们。

Chapter 3, Unleashing Machine Learning Potentials in NLP, discusses different concepts and methods in ML that can be used to tackle NLP problems. We will discuss general feature selection and classification techniques. We will cover general aspects of ML problems, such as train/test/validation selection, and dealing with imbalanced datasets. We will also discuss performance metrics for evaluating ML models that are used in NLP problems. We will explain the theory behind the methods as well as how to use them in code.

第 4 章，简化文本预处理技术以实现最佳 NLP 性能，讨论现实世界问题背景下的各种文本预处理步骤。我们将根据要解决的场景解释哪些步骤适合哪些需求。本章将介绍和回顾一个完整的 Python 管道

Chapter 4, Streamlining Text Preprocessing Techniques for Optimal NLP Performance, talks about various text preprocessing steps in the context of real-world problems. We will explain which steps suit which needs, based on the scenario that is to be solved. There will be a complete Python pipeline presented and reviewed in this chapter.

第 5 章，增强文本分类能力：利用传统机器学习技术，解释如何执行文本分类。还将解释理论和实现。一个全面的 Python 笔记本将作为案例研究进行介绍。

Chapter 5, Empowering Text Classification: Leveraging Traditional Machine Learning Techniques, explains how to perform text classification. Theory and implementation will also be explained. A comprehensive Python notebook will be covered as a case study.

第 6 章，重新构想文本分类：深入研究深度学习语言模型，涵盖了可以使用深度学习神经网络解决的问题。我们将向您介绍此类别中的不同问题，以便您学习如何有效地解决它们。这里将解释这些方法的理论，并将涵盖一个全面的 Python 笔记本作为案例研究。

Chapter 6, Text Classification Reimagined: Delving Deep into Deep Learning Language Models, covers the problems that can be solved using deep learning neural networks. The different problems in this category will be introduced to you so you can learn how to efficiently solve them. The theory of the methods will be explained here and a comprehensive Python notebook will be covered as a case study.

第 7 章，揭秘大型语言模型：理论、设计和 Langchain 实现，概述了LLMs开发和使用背后的动机，以及其创建过程中面临的挑战。通过检查最先进的模型设计，您将全面了解LLMs的理论基础和实际应用。

Chapter 7, Demystifying Large Language Models: Theory, Design, and Langchain Implementation, outlines the motivations behind the development and usage of LLMs, alongside the challenges faced during their creation. Through an examination of state-of-the-art model designs, you will gain comprehensive insights into the theoretical underpinnings and practical applications of LLMs.

第 8 章，访问大型语言模型的力量：高级设置和与 RAG 集成，指导您设置基于 API 和开源的 LLM 应用程序，并通过 LangChain 深入研究即时工程和 RAG。我们将回顾代码中的实际应用。

Chapter 8, Accessing the Power of Large Language Models: Advanced Setup and Integration with RAG, guides you through setting up LLM applications, both API-based and open source, and delves into prompt engineering and RAGs via LangChain. We will review practical applications in code.

第 9 章，探索前沿：LLM 驱动的高级应用和创新，深入探讨使用 RAG 提高 LLM 性能，探索先进方法、自动 Web 源检索、即时压缩、API 成本降低和协作多代理 LLM 团队，推动当前LLM申请的边界。在这里，您将查看多个 Python 笔记本，每个笔记本处理实际用例的不同高级解决方案。

Chapter 9, Exploring the Frontiers: Advanced Applications and Innovations Driven by LLMs, dives into enhancing LLM performance using RAG, exploring advanced methodologies, automatic web source retrieval, prompt compression, API-cost reduction, and collaborative multi-agent LLM teams, pushing the boundaries of current LLM applications. Here, you will review multiple Python notebooks, each handling different advanced solutions to practical use cases.

第 10 章，驾驭浪潮：分析LLMs和人工智能塑造的过去、现在和未来趋势，深入探讨LLMs和人工智能对技术、文化和社会的变革性影响，探索关键趋势、计算进步、大型数据集的重要性以及LLMs在商业及其他领域的演变、目的和社会影响。

Chapter 10, Riding the Wave: Analyzing Past, Present, and Future Trends Shaped by LLMs and AI, dives into the transformative impact of LLMs and AI on technology, culture, and society, exploring key trends, computational advancements, the significance of large datasets, and the evolution, purpose, and social implications of LLMs in business and beyond.

第 11 章，独家行业洞察：世界级专家的观点和预测，通过与法律、研究和执行角色专家的对话，深入探讨未来 NLP 和 LLM 的趋势，探索挑战、机遇以及 LLM 与专业人士的交叉点实践和道德考虑。

Chapter 11, Exclusive Industry Insights: Perspectives and Predictions from World Class Experts, offers a deep dive into future NLP and LLM trends through conversations with experts in legal, research, and executive roles, exploring challenges, opportunities, and the intersection of LLMs with professional practices and ethical considerations.

为了充分利用这本书

To get the most out of this book

本书中提供的所有代码均以 Jupyter Notebook 的形式呈现。所有代码均使用 Python 3.10.X 开发，预计也适用于更高版本。

All the code presented in this book is in the form of a Jupyter notebook. All the code was developed with Python 3.10.X and is expected to work on later versions as well.

由于本书中的代码示例具有多样化的用例，对于一些高级 LLM 解决方案，您将需要一个 OpenAI 帐户，该帐户将允许API 密钥。

As the code examples in this book have a diversified set of use cases, for some of the advanced LLM solutions, you will need an OpenAI account, which will allow an API key.

如果您使用的是本书的数字版本，我们建议您自己键入代码或从本书的 GitHub 存储库访问代码（下一节中提供了链接）。这样做将帮助您避免与复制和粘贴代码相关的任何潜在错误。

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

下载示例代码文件

Download the example code files

您可以从 GitHub 下载本书的示例代码文件：https://github.com/PacktPublishing/Mastering-NLP-from-Foundations-to-LLMs如果代码有更新，它将在GitHub 存储库中更新。

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Mastering-NLP-from-Foundations-to-LLMs If there’s an update to the code, it will be updated in the GitHub repository.

在整本书中，我们回顾了代表专业行业级别解决方案的完整代码笔记本：

Throughout the book we review complete code notebooks that represent solutions on a professional industry level:

我们还提供丰富的书籍和视频目录中的其他代码包，网址为https://github.com/PacktPublishing/。去看一下！

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

使用的约定

Conventions used

本书中使用了许多文本约定。

There are a number of text conventions used throughout this book.

文本中的代码：表示文本中的代码字、数据库表名称、文件夹名称、文件名、文件扩展名、路径名、虚拟 URL、用户输入和 Twitter 句柄。这是一个例子：“现在，我们添加一个用于实现语法的功能。我们定义了output_parser变量，并使用不同的函数来生成输出，predict_and_parse()。”

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Now, we add a feature for achieving the syntax. We define the output_parser variable, and we use a different function for generating the output, predict_and_parse().”

代码块设置如下：

A block of code is set as follows:

将 pandas 导入为 pd
将 matplotlib.pyplot 导入为 plt
# 从 URL 加载记录字典
导入请求
进口泡菜

import pandas as pd
import matplotlib.pyplot as plt
# Load the record dict from URL
import requests
import pickle

当我们希望引起您对代码块的特定部分的注意时，相关行或项目会设置为粗体：

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

qa_engineer（致 manager_0）：
exitcode: 0 (执行成功)
代码输出：
图(640x480)
程序员（对 manager_0）：
终止

qa_engineer (to manager_0):
exitcode: 0 (execution succeeded)
Code output:
Figure(640x480)
programmer (to manager_0):
TERMINATE

粗体：表示新术语、重要单词或您在屏幕上看到的单词。例如，菜单或对话框中的单词以粗体显示。下面是一个示例：“虽然我们选择了一个特定数据库，但您可以参考Vector Store页面以了解有关不同选择的更多信息。”

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “While we chose one particular database, you can refer to the Vector Store page to read more about the different choices.”

提示或重要说明

Tips or important notes

看起来像这样。

Appear like this.

保持联系

Get in touch

我们随时欢迎读者提供反馈。

Feedback from our readers is always welcome.

一般反馈：如果您对本书的任何方面有疑问，请发送电子邮件至customercare@packtpub.com ，并在邮件主题中提及书名。

General feedback: If you have questions about any aspect of this book, email us at customercare@packtpub.com and mention the book title in the subject of your message.

勘误表：尽管我们已尽一切努力确保内容的准确性，但错误还是会发生。如果您发现本书中有错误，请向我们报告，我们将不胜感激。请访问www.packtpub.com/support/errata并填写表格。

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

盗版：如果您在互联网上发现任何形式的非法复制品，请向我们提供位置地址或网站名称，我们将不胜感激。请通过copyright@packt.com联系我们并提供材料链接。

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material.

如果您有兴趣成为一名作家：如果您有某个主题的专业知识并且您有兴趣撰写或撰写书籍，请访问authors.packtpub.com。

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

评论

Reviews

请留下评论。一旦您阅读并使用了这本书，为什么不在您购买该书的网站上留下评论呢？然后，潜在读者可以看到并使用您的公正意见来做出购买决定，Packt 可以了解您对我们产品的看法，我们的作者可以看到您对他们的书的反馈。谢谢你！

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

有关 Packt 的更多信息，请访问www.packtpub.com。

For more information about Packt, please visit www.packtpub.com.

分享你的意见

Share Your Thoughts

一旦您阅读了《掌握 NLP 从基础到LLMs》，我们很想听听您的想法！请点击此处直接进入本书的亚马逊评论页面并分享您的反馈。

Once you’ve read Mastering NLP from Foundations to LLMs, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

您的评论对我们和技术社区都很重要，并将帮助我们确保提供优质的内容。

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

下载本书的免费 PDF 副本

Download a free PDF copy of this book

感谢您购买本书！

Thanks for purchasing this book!

您是否喜欢在旅途中阅读，但无法随身携带纸质书籍？

Do you like to read on the go but are unable to carry your print books everywhere?

您购买的电子书是否与您选择的设备不兼容？

Is your eBook purchase not compatible with the device of your choice?

不用担心，现在对于每本 Packt 书籍，您都可以免费获得该书的无 DRM 的 PDF 版本。

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

随时随地在任何设备上阅读。从您喜爱的技术书籍中搜索、复制代码并将其直接粘贴到您的应用程序中。

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

福利还不止于此，您每天都可以在收件箱中独家获得折扣、新闻通讯和精彩的免费内容

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

请遵循以下简单步骤即可获得好处：

Follow these simple steps to get the benefits:

1. 扫描二维码或访问以下链接
2. Scan the QR code or visit the link below

https://packt.link/free-ebook/978-1-80461-918-6

https://packt.link/free-ebook/978-1-80461-918-6

2. 提交您的购买证明
2. Submit your proof of purchase
3. 就是这样！我们会将您的免费 PDF 和其他福利直接发送到您的电子邮件
4. That’s it! We’ll send your free PDF and other benefits to your email directly

1

1

探索 NLP 领域：全面介绍

Navigating the NLP Landscape: A Comprehensive Introduction

本书旨在帮助专业人士将自然语言处理（NLP）技术应用到他们的工作中，无论他们是从事 NLP 项目还是在数据科学等其他领域使用 NLP。本书的目的是向您介绍 NLP 领域及其底层技术，包括机器学习( ML ) 和深度学习( DL )。在整本书中，我们强调了数学基础的重要性，例如线性代数、统计和概率以及优化理论，这些对于理解 NLP 中使用的算法是必要的。内容附有 Python 代码示例，允许您预先练习、实验并生成书中介绍的一些开发内容。

This book is aimed at helping professionals apply natural language processing (NLP) techniques to their work, whether they are working on NLP projects or using NLP in other areas, such as data science. The purpose of the book is to introduce you to the field of NLP and its underlying techniques, including machine learning (ML) and deep learning (DL). Throughout the book, we highlight the importance of mathematical foundations, such as linear algebra, statistics and probability, and optimization theory, which are necessary to understand the algorithms used in NLP. The content is accompanied by code examples in Python to allow you to pre-practice, experiment, and generate some of the development presented in the book.

本书讨论了 NLP 面临的挑战，例如理解单词的上下文和含义、它们之间的关系以及对标记数据的需求。书中还提到了 NLP 的最新进展，包括预训练的语言模型，例如 BERT 和 GPT，以及大量文本数据的可用性，这些都提高了NLP 任务的性能。

The book discusses the challenges faced in NLP, such as understanding the context and meaning of words, the relationships between them, and the need for labeled data. The book also mentions the recent advancements in NLP, including pre-trained language models, such as BERT and GPT, and the availability of large amounts of text data, which has led to improved performance on NLP tasks.

本书将通过讨论语言模型对 NLP 领域的影响来吸引您，包括提高 NLP 任务的准确性和有效性、开发更先进的 NLP 系统以及更广泛的人群的可访问性。

The book will engage you by discussing the impact of language models on the field of NLP, including improved accuracy and effectiveness in NLP tasks, the development of more advanced NLP systems, and accessibility to a broader range of people.

我们将在本章中介绍以下标题：

We will be covering the following headings in the chapter:

• 什么是自然语言处理？
• What is natural language processing?
• 自然语言机器处理的初始策略
• Initial strategies in the machine processing of natural language
• 获胜的协同效应——NLP和 ML的结合
• A winning synergy – the coming together of NLP and ML
• NLP 中的数学和统计学简介
• Introduction to math and statistics in NLP

这本书适合谁

Who this book is for

本书的目标读者是在项目中使用文本的专业人士。这可能包括 NLP 从业者，他们可能是初学者，以及那些通常不使用文本的人。

The target audience of the book is professionals who work with text as part of their projects. This may include NLP practitioners, who may be beginners, as well as those who do not typically work with text.

什么是自然语言处理？

What is natural language processing?

NLP是人工智能( AI )的一个领域，专注于计算机与人类之间的交互语言。它涉及使用计算理解、解释和生成人类语言的技术，使计算机能够自然且有意义地理解和响应人类输入。

NLP is a field of artificial intelligence (AI) focused on the interaction between computers and human languages. It involves using computational techniques to understand, interpret, and generate human language, making it possible for computers to understand and respond to human input naturally and meaningfully.

自然语言处理的历史和演变

The history and evolution of natural language processing

NLP 的历史非常精彩穿越时间的旅程，可以追溯到 20 世纪 50 年代，艾伦·图灵等先驱做出了重大贡献。图灵的开创性论文《计算机器与智能》介绍了图灵测试，为未来人工智能和自然语言处理的探索奠定了基础。这一时期标志着符号 NLP 的开始，其特点是使用基于规则的系统，例如 1954 年著名的乔治城实验，该实验雄心勃勃地旨在通过将俄语内容翻译成英语来解决机器翻译问题（参见https:// en.wikipedia.org/wiki/Georgetown%E2%80%93IBM_experiment）。尽管早期很乐观，但进展缓慢，揭示了语言理解和生成的复杂性。

The history of NLP is a fascinating journey through time, tracing back to the 1950s, with significant contributions from pioneers such as Alan Turing. Turing’s seminal paper, Computing Machinery and Intelligence, introduced the Turing test, laying the groundwork for future explorations in AI and NLP. This period marked the inception of symbolic NLP, characterized by the use of rule-based systems, such as the notable Georgetown experiment in 1954, which ambitiously aimed to solve machine translation by generating a translation of Russian content into English (see https://en.wikipedia.org/wiki/Georgetown%E2%80%93IBM_experiment). Despite early optimism, progress was slow, revealing the complexities of language understanding and generation.

20 世纪 60 年代和 1970 年代见证了早期 NLP 系统的发展，该系统展示了机器使用有限的词汇和知识库进行类人交互的潜力。这个时代还见证了概念本体的创建，这对于以计算机可理解的格式构建现实世界信息至关重要。然而，在 ML 的进步和计算能力的增强的推动下，基于规则的方法的局限性导致了 20 世纪 80 年代末向统计 NLP 的范式转变。这一转变使得能够更有效地从大型语料库中学习，从而显着推进机器翻译和其他 NLP 任务。这种范式转变不仅代表了技术和方法论的进步，而且强调了 NLP 中语言学方法的概念演变。在摆脱预定义语法规则的僵化过程中，这种转变采用了语料库语言学，这种方法允许机器通过广泛接触大量文本来“感知”和理解语言。这种方法反映了对语言更加实证和数据驱动的理解，其中模式和含义源自实际语言使用而不是理论构造，从而实现更细致、更灵活的语言处理能力。

The 1960s and 1970s saw the development of early NLP systems, which demonstrated the potential for machines to engage in human-like interactions using limited vocabularies and knowledge bases. This era also witnessed the creation of conceptual ontologies, crucial for structuring real-world information in a computer-understandable format. However, the limitations of rule-based methods led to a paradigm shift in the late 1980s towards statistical NLP, fueled by advances in ML and increased computational power. This shift enabled more effective learning from large corpora, significantly advancing machine translation and other NLP tasks. This paradigm shift not only represented a technological and methodological advancement but also underscored a conceptual evolution in the approach to linguistics within NLP. In moving away from the rigidity of predefined grammar rules, this transition embraced corpus linguistics, a method that allows machines to “perceive” and understand languages through extensive exposure to large bodies of text. This approach reflects a more empirical and data-driven understanding of language, where patterns and meanings are derived from actual language use rather than theoretical constructs, enabling more nuanced and flexible language processing capabilities.

进入21世纪，网络的出现提供了大量数据，催化了无监督和半监督学习算法的研究。随着 2010 年代神经 NLP 的出现，这一突破随之而来，深度学习技术开始占据主导地位，在语言建模和解析方面提供了前所未有的准确性。这个时代的标志是 Word2Vec 等复杂模型的发展和深度神经网络的激增，推动 NLP 走向更自然、更有效的人机交互。随着我们继续在这些进步的基础上发展，NLP 站在了人工智能研究的最前沿，其历史反映了对理解和复制人类语言细微差别的不懈追求。

Entering the 21st century, the emergence of the web provided vast amounts of data, catalyzing research in unsupervised and semi-supervised learning algorithms. The breakthrough came with the advent of neural NLP in the 2010s, where DL techniques began to dominate, offering unprecedented accuracy in language modeling and parsing. This era has been marked by the development of sophisticated models such as Word2Vec and the proliferation of deep neural networks, driving NLP towards more natural and effective human-computer interaction. As we continue to build on these advancements, NLP stands at the forefront of AI research, with its history reflecting a relentless pursuit of understanding and replicating the nuances of human language.

近年来，NLP也被广泛应用于医疗、金融、社交媒体等行业，用于自动化决策、增强人与机器之间的沟通。例如，NLP 已被用于从医疗文档中提取信息、分析客户反馈、在语言之间翻译文档以及搜索大量帖子。

In recent years, NLP has also been applied to a wide range of industries, such as healthcare, finance, and social media, where it has been used to automate decision-making and enhance communication between humans and machines. For example, NLP has been used to extract information from medical documents, analyze customer feedback, translate documents between languages, and search through enormous amounts of posts.

自然语言机器处理的初始策略

Initial strategies in the machine processing of natural language

NLP 中的传统方法包括文本预处理，它与文本准备同义，即文本预处理。然后应用机器学习方法。预处理文本是 NLP 和 ML 应用程序中的重要步骤。它涉及将原始文本数据清理并转换为机器学习算法可以轻松理解和分析的形式。预处理的目标是消除噪声和不一致并标准化数据，使其更适合高级 NLP 和ML 方法。

Traditional methods in NLP consist of text preprocessing, which is synonymous with text preparation, which is then followed by applying ML methods. Preprocessing text is an essential step in NLP and ML applications. It involves cleaning and transforming the original text data into a form that can be easily understood and analyzed by ML algorithms. The goal of preprocessing is to remove noise and inconsistencies and standardize the data, making it more suitable for advanced NLP and ML methods.

预处理的主要好处之一是它可以显着提高机器学习算法的性能。例如，删除停用词（例如“the”和“is”等没有太多意义的常用词）可以帮助降低数据的维度，使算法更容易识别模式。

One of the key benefits of preprocessing is that it can significantly improve the performance of ML algorithms. For example, removing stop words, which are common words that do not carry much meaning, such as “the” and “is,” can help reduce the dimensionality of the data, making it easier for the algorithm to identify patterns.

以下面的句子为例：

Take the following sentence as an example:

我要去商店买一些牛奶和面包。

I am going to the store to buy some milk and bread.

删除停用词后，我们有以下内容：

After removing the stop words, we have the following:

去商店买牛奶面包。

going store buy milk bread.

在例句中，停用词“ I ”、“ am ”、“ to ”、“ the ”、“ some ”和“ and ”不会给句子添加任何附加含义，可以在不改变整体含义的情况下删除。的句子。应该强调的是，停用词的删除需要根据具体目标进行调整，因为省略特定单词在一种情况下可能微不足道，但在另一种情况下却是有害的。

In the example sentence, the stop words “I,” “am,” “to,” “the,” “some,” and “and” do not add any additional meaning to the sentence and can be removed without changing the overall meaning of the sentence. It should be emphasized that the removal of stop words needs to be tailored to the specific objective, as the omission of a particular word might be trivial in one context but detrimental in another.

此外，词干提取和词形还原可以减少单词的基本形式，可以帮助减少数据中唯一单词的数量，使算法更容易识别它们之间的关系，这将在本书中完整解释。

Additionally, stemming and lemmatization, which reduce words to their base forms, can help reduce the number of unique words in the data, making it easier for the algorithm to identify relationships between them, which will be explained completely in this book.

以下面的句子为例：

Take the following sentence as an example:

孩子们跑得快、跳得快、游得快。

The boys ran, jumped, and swam quickly.

应用词干提取（将每个单词简化为其词根或词干形式）后，不考虑单词时态或派生词缀，我们可能会得到：

After applying stemming, which reduces each word to its root or stem form, disregarding word tense or derivational affixes, we might get:

男孩跑得快、跳得快、游得快。

The boy ran, jump, and swam quick.

词干提取将文本简化为其基本形式。在此示例中，“ ran ”、“ jumped ”和“ swam ”分别简化为“ ran ”、“ jump ”和“ swam ”。请注意，“ran”和“swam”不会改变，因为词干提取通常会产生接近其词根形式但不完全是字典基本形式的单词。此过程有助于降低文本数据的复杂性，使机器学习算法更容易匹配和分析模式，而不会因同一单词的变化而陷入困境。

Stemming simplifies the text to its base forms. In this example, “ran,” “jumped,” and “swam” are reduced to “ran,” “jump,” and “swam,” respectively. Note that “ran” and “swam” do not change, as stemming often results in words that are close to their root form but not exactly the dictionary base form. This process helps reduce the complexity of the text data, making it easier for machine learning algorithms to match and analyze patterns without getting bogged down by variations of the same word.

以下面的句子为例：

Take the following sentence as an example:

孩子们跑得快、跳得快、游得快。

The boys ran, jumped, and swam quickly.

应用词形还原（考虑单词的形态分析）后，旨在返回单词的基本形式或字典形式（称为引理），我们得到：

After applying lemmatization, which considers the morphological analysis of the words, aiming to return the base or dictionary form of a word, known as the lemma, we get:

这个男孩跑得快、跳得快、游得快。

The boy run, jump, and swim quickly.

词形还原准确地将“ ran ”、“ jumped ”和“ swam ”转换为“ run ”、“ jump ”和“ swim ”。此过程考虑了每个单词的词性，确保还原为基本形式在语法和上下文上都是适当的。与词干提取不同，词形还原可以更精确地还原基本形式，从而确保处理后的文本仍然有意义并且上下文准确。这使 NLP 模型能够更有效地理解和处理语言，从而提高了 NLP 模型的性能，降低了数据集的复杂性，同时保持了原始文本的完整性。

Lemmatization accurately converts “ran,” “jumped,” and “swam” to “run,” “jump,” and “swim.” This process takes into account the part of speech of each word, ensuring that the reduction to the base form is both grammatically and contextually appropriate. Unlike stemming, lemmatization provides a more precise reduction to the base form, ensuring that the processed text remains meaningful and contextually accurate. This enhances the performance of NLP models by enabling them to understand and process language more effectively, reducing the dataset’s complexity while maintaining the integrity of the original text.

预处理的另外两个重要方面是数据标准化和数据清理。数据规范化包括将所有文本转换为小写、删除标点符号以及标准化数据格式。这有助于确保算法不会将同一单词的不同变体视为单独的实体，这可能导致结果不准确。

Two other important aspects of preprocessing are data normalization and data cleaning. Data normalization includes converting all text to lowercase, removing punctuation, and standardizing the format of the data. This helps to ensure that the algorithm does not treat different variations of the same word as separate entities, which can lead to inaccurate results.

数据清理包括删除重复或不相关的数据，并纠正数据中的错误或不一致。这在大型数据集中尤其重要，因为手动清理既耗时又容易出错。自动化预处理工具可以帮助快速识别和消除错误，使数据分析更加可靠。

Data cleaning includes removing duplicate or irrelevant data and correcting errors or inconsistencies in the data. This is particularly important in large datasets, where manual cleaning is time-consuming and error-prone. Automated preprocessing tools can help to quickly identify and remove errors, making the data more reliable for analysis.

图 1 .1描绘了一个全面的预处理流程。我们将在第 4 章中介绍此代码示例：

Figure 1.1 portrays a comprehensive preprocessing pipeline. We will cover this code example in Chapter 4:

图 1.1 – 综合预处理流程

Figure 1.1 – Comprehensive preprocessing pipeline

总之，文本预处理是 NLP 和 ML 应用中至关重要的一步；它通过消除噪声和不一致以及标准化数据来提高机器学习算法的性能。此外，它在 NLP 任务的数据准备和数据清理。通过在预处理上投入时间和资源，可以确保数据的高质量，并为先进的 NLP 和 ML 方法做好准备，从而获得更准确、更可靠的结果。

In conclusion, preprocessing text is a vital step in NLP and ML applications; it improves the performance of ML algorithms by removing noise and inconsistencies and standardizing the data. Additionally, it plays a crucial role in data preparation for NLP tasks and in data cleaning. By investing time and resources in preprocessing, one can ensure that the data is of high quality and is ready for advanced NLP and ML methods, resulting in more accurate and reliable results.

当我们的文本数据准备好进一步处理时，下一步通常涉及对其拟合 ML 模型。

As our text data is prepared for further processing, the next step typically involves fitting an ML model to it.

获胜的协同效应——NLP 和 ML 的结合

A winning synergy – the coming together of NLP and ML

机器学习是人工智能的一个子领域，涉及训练从数据中学习的算法，使它们能够做出预测或决策，而无需这些数据明确编程。ML 正在推动许多不同领域的进步，例如计算机视觉、语音识别，当然还有NLP。

ML is a subfield of AI that involves training algorithms to learn from data, allowing them to make predictions or decisions without those being explicitly programmed. ML is driving advancements in so many different fields, such as computer vision, voice recognition, and, of course, NLP.

进一步深入了解 ML 的具体技术，NLP 中使用的一项特殊技术是统计语言建模，它涉及在大型文本语料库上训练算法来预测给定单词序列的可能性。它被用于广泛的应用，例如语音识别、机器翻译和文本生成。

Diving a little more into the specific techniques of ML, a particular technique used in NLP is statistical language modeling, which involves training algorithms on large text corpora to predict the likelihood of a given sequence of words. This is used in a wide range of applications, such as speech recognition, machine translation, and text generation.

另一项重要技术是深度学习，它是机器学习的一个子领域，涉及训练人工神经网络网络上的大量数据。DL 模型，例如卷积神经网络( CNN ) 和循环神经网络( RNN )，已事实证明，它足以胜任语言理解、文本摘要和情感分析等 NLP 任务。

Another essential technique is DL, which is a subfield of ML that involves training artificial neural networks on large amounts of data. DL models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have been shown to be adequate for NLP tasks such as language understanding, text summarization, and sentiment analysis.

图 1 .2描绘了 AI、ML、DL和 NLP之间的关系

Figure 1.2 portrays the relationship between AI, ML, DL, and NLP:

图 1.2 – 不同学科之间的关系

Figure 1.2 – The relationship between the different disciplines

NLP 中的数学和统计学简介

Introduction to math and statistics in NLP

NLP 和 ML 的坚实基础是算法的数学基础。特别是，关键基础是线性代数、统计和概率以及优化理论。第 2 章将调查了解这些主题所需的关键主题。在整本书中，我们将为各种方法和假设提供证明和理由。

The solid base for NLP and ML is the mathematical foundations from which the algorithms stem. In particular, the key foundations are linear algebra, statistics and probability, and optimization theory. Chapter 2 will survey the key topics you will need to understand these topics. Throughout the book, we will present proofs and justifications for the various methods and hypotheses.

NLP 的挑战之一是处理以人类语言生成的大量数据。这包括理解上下文、单词的含义以及它们之间的关系。为了应对这一挑战，研究人员开发了各种技术，例如嵌入和注意力机制，它们分别以数字格式表示单词的含义，并帮助识别文本中最关键的部分。

One of the challenges in NLP is dealing with the vast amount of data that is generated in human language. This includes understanding the context, as well as the meaning of the words and relationships between them. To deal with this challenge, researchers have developed various techniques, such as embeddings and attention mechanisms, which represent the meaning of words in a numerical format and help identify the most critical parts of the text, respectively.

NLP 的另一个挑战是需要标记数据，因为手动注释大型文本语料库既昂贵又耗时。为了解决这个问题，研究人员开发了无监督和弱监督方法，可以从未标记的数据中学习，例如聚类、主题建模和自监督学习。

Another challenge in NLP is the need for labeled data, as manually annotating large text corpora is expensive and time-consuming. To address this problem, researchers have developed unsupervised and weakly supervised methods that can learn from unlabeled data, such as clustering, topic modeling, and self-supervised learning.

总体而言，NLP 是一个快速发展的领域，有潜力改变我们与计算机交互的方式信息。它用于各种应用程序，从聊天机器人到语言翻译为文本摘要和情感分析。统计语言建模和深度学习等机器学习技术的使用对于开发这些系统至关重要。正在进行的研究解决了剩余的挑战，例如理解上下文和处理标记数据的缺乏。

Overall, NLP is a rapidly evolving field that has the potential to transform the way we interact with computers and information. It is used in various applications, from chatbots and language translation to text summarization and sentiment analysis. The use of ML techniques, such as statistical language modeling and DL, has been crucial in developing these systems. Ongoing research addresses the remaining challenges, such as understanding context and dealing with the lack of labeled data.

最重要的之一NLP 的进步在于预训练语言模型的开发，例如来自 Transformer ( BERT ) 的双向编码器表示和生成式预训练 Transformer ( GPT )。这些模型已经过大量训练文本数据，并且可以针对特定任务进行微调，例如情感分析或语言翻译。

One of the most significant advances in NLP has been the development of pre-trained language models, such as bidirectional encoder representations from transformers (BERTs) and generative pre-trained transformers (GPTs). These models have been trained on massive amounts of text data and can be fine-tuned for specific tasks, such as sentiment analysis or language translation.

Transformer 是 BERT 和 GPT 模型背后的技术，它使机器能够更有效地理解句子中单词的上下文，从而彻底改变了 NLP。与以前线性处理文本的方法不同，Transformer可以并行处理单词，通过注意机制捕获语言中的细微差别。这使他们能够辨别每个单词相对于其他单词的重要性，极大地增强了模型掌握复杂语言模式和细微差别的能力，并为 NLP 应用的准确性和流畅性设立了新标准。这增强了 NLP 应用程序的创建，并提高了各种NLP 任务的性能。

Transformers, the technology behind the BERT and GPT models, revolutionized NLP by enabling machines to understand the context of words in sentences more effectively. Unlike previous methods that processed text linearly, transformers can handle words in parallel, capturing nuances in language through attention mechanisms. This allows them to discern the importance of each word relative to others, greatly enhancing the model’s ability to grasp complex language patterns and nuances and setting a new standard for accuracy and fluency in NLP applications. This has enhanced the creation of NLP applications and has led to improved performance on a wide range of NLP tasks.

图 1 .3详细介绍了 Transformer 组件的功能设计。

Figure 1.3 details the functional design of the Transformer component.

图 1.3 – 模型架构中的 Transformer

Figure 1.3 – Transformer in model architecture

NLP 的另一个重要发展是大量带注释文本数据的可用性增加，这使得可以训练更准确的模型。此外，无监督和半监督学习的发展技术允许在少量标记数据上训练模型，从而使 NLP 在更广泛的场景中应用成为可能。

Another important development in NLP has been the increase in the availability of large amounts of annotated text data, which has allowed for the training of more accurate models. Additionally, the development of unsupervised and semi-supervised learning techniques has allowed for the training of models on smaller amounts of labeled data, making it possible to apply NLP in a wider range of scenarios.

语言模型对 NLP 领域产生了重大影响。语言模型改变该领域的关键方式之一是提高自然语言处理任务的准确性和有效性。例如，许多语言模型都经过大量文本数据的训练，使它们能够更好地理解人类语言的细微差别和复杂性。这提高了语言翻译、文本摘要和情感分析等任务的性能。

Language models have had a significant impact on the field of NLP. One of the key ways that language models have changed the field is by improving the accuracy and effectiveness of natural language processing tasks. For example, many language models have been trained on large amounts of text data, allowing them to better understand the nuances and complexities of human language. This has led to improved performance in tasks such as language translation, text summarization, and sentiment analysis.

语言模型改变 NLP 领域的另一种方式是支持更高级、更复杂的 NLP 的开发系统。例如，一些语言模型，例如GPT，可以生成类似人类的文本，这为自然语言生成和对话系统开辟了新的可能性。其他语言模型，例如 BERT，提高了问答、情感分析和命名实体识别等任务的性能。

Another way that language models have changed the field of NLP is by enabling the development of more advanced, sophisticated NLP systems. For example, some language models, such as GPT, can generate human-like text, which has opened up new possibilities for natural language generation and dialogue systems. Other language models, such as BERT, have improved the performance of tasks such as question answering, sentiment analysis, and named entity recognition.

语言模型也改变了这个领域，让更广泛的人更容易接触到它。随着预训练语言模型的出现，开发人员现在可以轻松地将这些模型微调到特定任务，而无需大量标记数据或从头开始训练模型的专业知识。这使得开发人员更容易构建 NLP 应用程序，并导致基于 NLP 的新产品和服务的爆炸式增长。

Language models have also changed the field by making it more accessible to a broader range of people. With the advent of pre-trained language models, developers can now easily fine-tune these models to specific tasks without the need for large amounts of labeled data or the expertise to train models from scratch. This has made it easier for developers to build NLP applications and has led to an explosion of new NLP-based products and services.

总体而言，语言模型通过提高现有 NLP 任务的性能、支持开发更先进的 NLP 系统以及使 NLP 更容易被更广泛的人群使用，在推动 NLP 领域发挥了关键作用。

Overall, language models have played a key role in advancing the field of NLP by improving the performance of existing NLP tasks, enabling the development of more advanced NLP systems, and making NLP more accessible to a broader range of people.

理解语言模型 – ChatGPT 示例

Understanding language models – ChatGPT example

ChatGPT 是 GPT 模型的变体，因其能够生成类人文本而变得流行，可用于广泛的自然语言生成任务，例如聊天机器人系统、文本摘要和对话系统。

ChatGPT, a variant of the GPT model, has become popular because of its ability to generate human-like text, which can be used for a broad range of natural language generation tasks, such as chatbot systems, text summarization, and dialogue systems.

其受欢迎的主要原因是其高质量的输出以及生成难以与人类编写的文本区分开的文本的能力。这使得它非常适合需要听起来自然的文本，例如聊天机器人系统、虚拟助手和文本摘要。

The main reason for its popularity is its high-quality outputs and its ability to generate text that is hard to distinguish from text written by humans. This makes it well-suited for applications that require natural-sounding text, such as chatbot systems, virtual assistants, and text summarization.

此外，ChatGPT 经过大量文本数据的预训练，使其能够理解人类语言的细微差别和复杂性。这使得它非常适合需要深入理解语言的应用程序，例如问答和情感分析。

Additionally, ChatGPT is pre-trained on a large amount of text data, allowing it to understand human language nuances and complexities. This makes it well-suited for applications that require a deep understanding of language, such as question answering and sentiment analysis.

此外，ChatGPT 可以通过为其提供少量特定于任务的数据来针对特定用例进行微调，这使其具有多功能性并适用于广泛的应用程序。它广泛应用于工业、研究和个人项目，范围包括客户服务聊天机器人、虚拟助理、自动化内容创建、文本摘要、对话系统、问答和情感分析。

Moreover, ChatGPT can be fine-tuned for specific use cases by providing it with a small amount of task-specific data, which makes it versatile and adaptable to a wide range of applications. It is widely used in industry, research, and personal projects, ranging from customer service chatbots, virtual assistants, automated content creation, text summarization, dialogue systems, question answering, and sentiment analysis.

总体而言，ChatGPT 生成高质量、类人文本的能力以及针对特定任务进行微调的能力使其成为各种自然语言生成应用程序的流行选择。

Overall, ChatGPT’s ability to generate high-quality, human-like text and its ability to be fine-tuned for specific tasks makes it a popular choice for a wide range of natural language generation applications.

现在让我们继续总结本章。

Let’s move on to summarize the chapter now.

概括

Summary

在本章中，我们向您介绍了 NLP 领域，它是人工智能的一个子领域。本章强调了数学基础的重要性，例如线性代数、统计和概率以及优化理论，这些对于理解 NLP 中使用的算法是必要的。它还涵盖了 NLP 中面临的挑战，例如理解单词的上下文和含义、它们之间的关系以及标记数据的需求。我们讨论了 NLP 的最新进展，包括预训练的语言模型，例如 BERT 和 GPT，以及大量文本数据的可用性，这导致了 NLP 任务性能的提高。当您了解文本预处理中数据清理、数据标准化、词干提取和词形还原的重要性时，我们谈到了文本预处理的重要性。然后我们讨论了 NLP 和 ML 的结合如何推动该领域的进步，并如何成为自动化任务和改善人机交互的日益重要的工具。

In this chapter, we introduced you to the field of NLP, which is a subfield of AI. The chapter highlights the importance of mathematical foundations, such as linear algebra, statistics and probability, and optimization theory, which are necessary to understand the algorithms used in NLP. It also covers the challenges faced in NLP, such as understanding the context and meaning of words, the relationships between them, and the need for labeled data. We discussed the recent advancements in NLP, including pre-trained language models, such as BERT and GPT, and the availability of large amounts of text data, which has led to improved performance in NLP tasks. We touched on the importance of text preprocessing as you gains knowledge of the importance of data cleaning, data normalization, stemming, and lemmatization in text preprocessing. We then talked about how the coming together of NLP and ML is driving advancements in the field and is becoming an increasingly important tool for automating tasks and improving human-computer interaction.

学习完本章后，您将能够理解 NLP、ML 和 DL 技术的重要性。您将能够了解 NLP 的最新进展，包括预训练的语言模型。您还将了解文本预处理的重要性以及它如何在 NLP 任务的数据准备和数据清理中发挥关键作用。

After learning from this chapter, you will be able to understand the importance of NLP, ML, and DL techniques. you will be able to understand the recent advancements in NLP, including pre-trained language models. you will also have gained knowledge of the importance of text preprocessing and how it plays a crucial role in data preparation for NLP tasks and in data cleaning.

在下一章中，我们将介绍机器学习的数学基础。这些基础将为我们提供整本书的帮助。

In the next chapter, we will cover the mathematical foundations of ML. These foundations will serve us throughout the book.

问题与解答

Questions and answers

1. 什么是自然语言处理（NLP）？
   • 问：NLP在人工智能领域的定义是什么？
   • 答：NLP 是人工智能的一个子领域，致力于使计算机能够以对人类用户来说既自然又有意义的方式理解、解释和生成人类语言。
2. What is natural language processing (NLP)?
   • Q: What defines NLP in the field of artificial intelligence?
   • A: NLP is a subfield of AI focused on enabling computers to understand, interpret, and generate human language in a way that is both natural and meaningful to human users.
3. 自然语言机器处理的初始策略。
   • 问： NLP 中预处理的重要性是什么？
   • 答：预处理，包括删除停用词、应用词干提取或词形还原等任务，对于清理和准备文本数据至关重要，从而提高机器学习算法在NLP 任务上的性能。
4. Initial strategies in machine processing of natural language.
   • Q: What is the importance of preprocessing in NLP?
   • A: Preprocessing, including tasks such as removing stop words and applying stemming or lemmatization, is crucial for cleaning and preparing text data, thereby improving the performance of machine learning algorithms on NLP tasks.
5. NLP 和机器学习 (ML)的协同作用。
   • 问：机器学习如何促进NLP 的进步？
   • 答：ML，尤其是统计语言建模和深度学习等技术，通过使算法能够从数据中学习、预测单词序列以及更有效地执行语言理解和情感分析等任务，推动 NLP 向前发展。
6. The synergy of NLP and machine learning (ML).
   • Q: How does machine learning contribute to advancements in NLP?
   • A: ML, especially techniques such as statistical language modeling and deep learning, drives NLP forward by enabling algorithms to learn from data, predict word sequences, and perform tasks such as language understanding and sentiment analysis more effectively.
7. NLP 中的数学和统计学简介
   • 问：为什么数学基础在 NLP 中很重要？
   • 答：线性代数、统计学和概率等数学基础对于理解和开发支持 NLP 技术（从基本预处理到复杂模型训练）的算法至关重要。
8. Introduction to math and statistics in NLP
   • Q: Why are mathematical foundations important in NLP?
   • A: Mathematical foundations such as linear algebra, statistics, and probability are essential for understanding and developing the algorithms that underpin NLP techniques, from basic preprocessing to complex model training.
9. NLP 的进步——预训练语言模型的作用
   • 问：BERT 和 GPT 等预训练模型对 NLP 有何影响？
   • 答：经过大量文本数据训练的预训练模型可以针对情感分析或语言翻译等特定任务进行微调，从而显着简化 NLP 应用程序的开发并提高任务性能。
10. Advancements in NLP – the role of pre-trained language models
    • Q: How have pre-trained models such as BERT and GPT influenced NLP?
    • A: Pre-trained models, trained on vast amounts of text data, can be fine-tuned for specific tasks such as sentiment analysis or language translation, significantly simplifying the development of NLP applications and enhancing task performance.
11. 理解语言模型中的Transformer
    • 问：为什么 Transformer 被认为是 NLP 的突破？
    • 答：Transformer 并行处理单词，并使用注意力机制来理解句子中的单词上下文，从而显着提高模型处理人类语言复杂性的能力。
12. Understanding transformers in language models
    • Q: Why are transformers considered a breakthrough in NLP?
    • A: Transformers process words in parallel and use attention mechanisms to understand word context within sentences, significantly improving a model’s ability to handle the complexities of human language.

2

2

掌握机器学习和 NLP 的线性代数、概率和统计

Mastering Linear Algebra, Probability, and Statistics for Machine Learning and NLP

自然语言处理( NLP ) 和机器学习( ML ) 是从数学概念（尤其是线性代数和概率论）中受益匪浅的两个领域。这些基本工具可以分析变量之间的关系，构成许多 NLP 和 ML 模型的基础。本章全面介绍线性代数和概率论，包括它们在 NLP 和 ML 中的实际应用。的c本章首先概述向量和矩阵，并涵盖基本运算。此外，还将解释理解后续章节中的概念和模型所需的统计学基础知识。最后，本章介绍了优化的基础知识，这对于解决 NLP 问题和理解变量之间的关系至关重要。读完本章后，您将在线性代数和概率论方面打下坚实的基础，并了解它们在 NLP和 ML 中的基本应用。

Natural language processing (NLP) and machine learning (ML) are two fields that have significantly benefited from mathematical concepts, particularly linear algebra and probability theory. These fundamental tools enable the analysis of the relationships between variables, forming the basis of many NLP and ML models. This chapter provides a comprehensive introduction to linear algebra and probability theory, including their practical applications in NLP and ML. The chapter commences with an overview of vectors and matrices and covers essential operations. Additionally, the basics of statistics, required for understanding the concepts and models in subsequent chapters, will be explained. Finally, the chapter introduces the fundamentals of optimization, which are critical for solving NLP problems and understanding the relationships between variables. By the end of this chapter, you will have a solid foundation in linear algebra and probability theory and understand their essential applications in NLP and ML.

在本章中，我们将讨论以下主题：

In this chapter, we’ll be covering the following topics:

• 线性代数简介
• Introduction to linear algebra
• 特征值和特征向量
• Eigenvalues and eigenvectors
• 机器学习的基本概率
• Basic probability for machine learning

线性代数简介

Introduction to linear algebra

让我们首先了解标量、向量和矩阵：

Let’s start by first understanding scalars, vectors, and matrices:

• 标量：标量是一个单一的通常来自真实的数值大多数机器学习应用程序中的域。NLP 中标量的示例包括文本语料库中单词的频率。
• Scalars: A scalar is a single numerical value that usually comes from the real domain in most ML applications. Examples of scalars in NLP include the frequency of a word in a text corpus.
• 向量：向量是数值元素的集合。这些元素中的每一个都可以称为条目、组件或维度，并且这些组件的计数定义了向量的维数。在 NLP 中，向量可以包含与词频、情感排名等元素相关的组件。NLP 和 ML 是从数学学科（尤其是线性代数和概率论）中获得巨大收益的两个领域。这些基础工具有助于评估变量之间的相关性，并且是众多 NLP 和 ML 模型的核心。本部分详细介绍了线性代数和概率论，以及它们在 NLP 和 ML 中的实际用法。例如，文本文档的三维向量表示可以表示为实数数组，例如[词频、情感排名、复杂度]。
• Vectors: A vector is a collection of numerical elements. Each of these elements can be termed as an entry, component, or dimension, and the count of these components defines the vector’s dimensionality. Within NLP, a vector could hold components related to elements such as word frequency, sentiment ranking, and more. NLP and ML are two domains that have reaped substantial benefits from mathematical disciplines, particularly linear algebra and probability theory. These foundational tools aid in evaluating the correlation between variables and are at the heart of numerous NLP and ML models. This segment presents a detailed primer on linear algebra and probability theory, along with their practical usage in NLP and ML. For instance, a text document’s three-dimensional vector representation might be expressed as a real-number array, such as [word frequency, sentiment ranking, complexity].
• 矩阵：矩阵可以被视为由行和列组成的数字元素的矩形集合。要从矩阵中检索元素，需要表示其行索引和列索引。在 NLP 领域，数据矩阵可能包括与不同文本文档对齐的行和与不同文本属性（例如词频、情感等）对齐的列。这种矩阵的维度由符号n × d表示，其中n是行数（即文本文档），d是列数（即属性）。
• Matrices: A matrix can be perceived as a rectangular collection of numerical elements composed of rows and columns. To retrieve an element from the matrix, one needs to denote its row and column indices. In the field of NLP, a data matrix might include rows that align with distinct text documents and columns that align with different text attributes, such as word frequency, sentiment, and so on. The dimensions of such a matrix are represented by the notation n × d, where n is the number of rows (i.e., text documents), and d is the number of columns (i.e., attributes).

接下来让我们继续讨论标量、向量和矩阵的基本运算。

Let’s move on to the basic operations for scalars, vectors, and matrices next.

标量、向量和矩阵的基本运算（加法和减法）可以在具有相同维度的向量上执行。让我们有两个向量：

The basic operations for scalars, vectors, and matrices—addition and subtraction—can be carried out on vectors with the same dimensions. Let’s have two vectors:

例如，如果我们有两个向量 a = [4,1] 和 b = [2,4]，则a + b = [ 6,5]。

For example, if we have two vectors, a = [4,1] and b = [2,4], then a + b = [6,5].

让我们将其想象为如下：

Let’s visualize this as follows:

图 2.1 – 两个向量相加（a = [4,1] 和 b = [2,4]）意味着 a + b = [6,5]

Figure 2.1 – Adding two vectors (a = [4,1] and b = [2,4]) means that a + b = [6,5]

可以通过将向量乘以标量来缩放向量。该运算是通过将向量的每个分量乘以标量值来执行的。例如，让我们考虑一个 n 维向量。将该向量缩放为 因子的过程可以用数学方式表示如下：

It is possible to scale a vector by multiplying it by a scalar. This operation is performed by multiplying each component of the vector by the scalar value. For example, let’s consider a n-dimensional vector, . The process of scaling this vector by a factor of can be represented mathematically as follows:

此操作会产生一个新向量，该向量与原始向量具有相同的维数，但每个分量都乘以标量值。

This operation results in a new vector that has the same dimensionality as the original vector but with each component multiplied by the scalar value .

有两种类型向量之间的乘法：点积 ( ) 和叉积 ( )。点积是我们在机器学习算法中经常使用的一种。

There are two types of multiplications between vectors: dot product () and cross product (). The dot product is the one we use often in ML algorithms.

点积是一种数学运算，可应用于两个向量x = [ x 1 , x 2 , … , x n ]和。它有许多实际应用，其中之一就是帮助确定它们的相似性。它被定义为两个向量的相应元素的乘积之和。x和y的点积由符号x y（中间有一个点）表示，定义如下：      

The dot product is a mathematical operation that can be applied to two vectors, x = [x 1, x 2, … , x n] and . It has many practical applications, one of which is to help determine their similarity. It is defined as the sum of the product of the corresponding elements of the two vectors. The dot product of x and y is represented by the symbol x y (having a dot in the middle) and is defined as follows:

其中n表示向量的维数。点积是一个标量，可用于测量两个向量之间的角度，以及一个向量到另一个向量的投影。它还在许多机器学习算法中发挥着重要作用，包括线性回归和神经网络。

where n represents the dimensionality of the vectors. The dot product is a scalar quantity and can be used to measure the angle between two vectors, as well as the projection of one vector onto another. It also serves a vital function in numerous ML algorithms, including linear regression and neural networks.

点积是可交换的，这意味着向量的顺序不会影响结果。这意味着x y = y x。此外，点积保持了标量乘法的分配性质，意味着以下内容：

The dot product is commutative, meaning that the order of the vectors does not affect the result. This means that x y = y x. Furthermore, the dot product maintains the distributive property of scalar multiplication, implying the following:

向量与其自身的点积也称为其平方范数或欧几里德范数。范数，用𝑛𝑜𝑟𝑚 (x)表示，表示向量的长度是 计算为

The dot product of a vector with itself is also known as its squared norm or Euclidean norm. The norm, symbolized by 𝑛𝑜𝑟𝑚(x), signifies the length of the vector and is computed as

向量的归一化可以通过将它们除以范数（也称为欧几里得范数或向量的长度）来实现。这会产生一个单位长度的向量，用x'表示。归一化过程可以表示为

The normalization of vectors can be achieved by dividing them by their norm, also known as the Euclidean norm or the length of the vector. This results in a vector with a unit length, denoted by x’. The normalization process can be shown as

其中是原始向量并表示其范数。应该注意的是，归一化向量具有在将其长度设置为 1 的同时保留其方向的效果，从而允许在不同空间中对向量进行有意义的比较。

where is the original vector and represents its norm. It should be noted that normalizing a vector has the effect of retaining its direction while setting its length to 1, allowing the meaningful comparison of vectors in different spaces.

两个向量之间的余弦相似度 在 数学上表示为两个向量标准化为单位长度后的点积。这可以写成如下：

The cosine similarity between two vectors and is mathematically represented as the dot product of the two vectors after they have been normalized to unit length. This can be written as follows:

其中和分别是向量x和y的范数。计算出的x和y之间的余弦相似度相当于两个向量之间的角度的余弦，表示为θ。

where and are the norms of the vectors x and y, respectively. This computed cosine similarity between x and y is equivalent to the cosine of the angle between the two vectors, denoted as θ.

点积为0的向量被视为正交，这意味着在具有两个非0向量的情况下，它们之间的角度为 90 度。我们可以得出结论，0向量与任何向量正交。如果一组向量的每一对都是正交的并且每个向量的范数为1 ，则该向量组被认为是正交的。这种正交集在许多数学环境中被证明是有价值的。例如，它们在不同正交坐标系之间转换时发挥作用，其中点的新坐标是相对于修改后的方向集计算的。这种方法在解析几何领域被称为坐标变换，它发现在线性代数领域有着广泛的应用。

Vectors with a dot product of 0 are deemed orthogonal, implying that in the case of having both non-0 vectors, the angle between them is 90 degrees. We can conclude that a 0 vector is orthogonal to any vector. A group of vectors is considered orthogonal if each pair of them is orthogonal and each vector possesses a norm of 1. Such orthonormal sets prove to be valuable in numerous mathematical contexts. For instance, they come into play when transforming between different orthogonal co-ordinate systems, where the new co-ordinates of a point are computed in relation to the modified direction set. This approach, known as co-ordinate transformation in the field of analytical geometry, finds widespread application in the realm of linear algebra.

矩阵和向量的基本运算

Basic operations on matrices and vectors

矩阵转置是获得矩阵转置的过程并涉及交换其行和列。这意味着原来位于矩阵中第(i, j)个位置的元素现在占据了其矩阵中第(j, i)个位置转置。结果，原来大小为n × m的矩阵转置后变为m × n矩阵。用于表示矩阵X转置的符号为。这是矩阵转置运算的说明性示例：

Matrix transpose is the process of obtaining the transpose of a matrix and involves interchanging its rows and columns. This means that the element originally at the (i, j)th position in the matrix now occupies the (j, i)th position in its transpose. As a result, a matrix that was originally of size n × m becomes an m × n matrix when transposed. The notation used to represent the transpose of matrix X is . Here’s an illustrative example of a matrix transposition operation:

至关重要的是，矩阵的转置恢复为原始矩阵X。此外，很明显，行向量可以转置为列向量，反之亦然。此外，以下内容对于矩阵和向量都成立：

Crucially, the transpose of matrix reverts to the original matrix X. Moreover, it is clear that row vectors can be transposed into column vectors and vice versa. Additionally, the following holds true for both matrices and vectors:

还值得注意的是，点积对于矩阵和向量是可交换的：

It’s also noteworthy that dot products are commutative for matrices and vectors:

矩阵定义

Matrix definitions

在本节中，我们将介绍不同类型的矩阵定义：

In this section, we’ll cover the different type of matrix definitions:

• 对称矩阵：对称矩阵是一种方阵，其中矩阵的转置等于原始矩阵。在用数学术语来说，如果矩阵X是对称的，则。 例如，

  

  是对称的。

• Symmetric matrix: A symmetric matrix is a type of square matrix where the transpose of the matrix is equal to the original matrix. In mathematical terms, if a matrix X is symmetric, then . For example,

  is symmetric.

• 矩形对角矩阵：这是一个维度为m × n且非0值的矩阵仅在主要对角线。
• Rectangular diagonal matrix: This is a matrix that is m × n in dimensions, with non-0 values only on the main diagonal.
• 上（或下）三角矩阵：如果一个矩阵的主对角线下方（上方）的所有元素（i，j）均为 0，则该矩阵称为上（三角）矩阵。接下来，我们将描述矩阵运营。
• Upper (or Lower) triangular matrix: A matrix is called an upper (triangular) matrix if all the entries (i,j) below (above) its main diagonal are 0. Next, we are going to describe matrix operations.

决定因素

Determinants

平方的行列式矩阵提供了乘以d维对象的坐标向量时对它的体积的影响的概念。行列式，符号为det(A)，表示由矩阵的行向量或列向量形成的平行六面体的（有符号）体积。这种解释始终成立，因为由行向量和列向量确定的体积在数学上是相同的。当可对角化矩阵A与一组坐标向量相互作用时，随之而来的失真称为各向异性缩放。行列式可以帮助建立此转换的比例因子。方阵的行列式对于通过与矩阵相乘实现的线性变化具有重要的见解。特别是，行列式的符号反映了基于系统方向的变换的影响。

The determinant of a square matrix provides a notion of its impact on the volume of a d-dimensional object when multiplied by its co-ordinate vectors. The determinant, symbolized as det(A), represents the (signed) volume of the parallelepiped formed by the row or column vectors of the matrix. This interpretation holds consistently, as the volume determined by the row and column vectors is mathematically identical. When a diagonalizable matrix A interacts with a group of co-ordinate vectors, the ensuing distortion is termed anisotropic scaling. The determinant can aid in establishing the scale factors of this conversion. The determinant of a square matrix carries crucial insights about the linear alteration accomplished by the multiplication with the matrix. Particularly, the sign of the determinant mirrors the impact of the transformation on the basis of the system’s orientation.

计算行列式为如下：

Calculating determinant is given as follows:

1. 对于1×1矩阵A，其行列式相当于其中存在的单个标量。
2. For a 1×1 matrix A, its determinant is equivalent to the single scalar present within it.
3. 对于较大的矩阵，可以通过固定一列j，然后使用该列中的元素进行扩展来计算行列式。作为另一种选择，可以修复行i并沿该特定行扩展。无论您是否选择修复行或列，最终结果（即矩阵的行列式）将保持一致。

   j为固定值，范围从1到d，

4. For larger matrices, the determinant can be calculated by securing a column, j, and then broadening using the elements within that column. As another option, it’s possible to fix a row, i, and expand along that particular row. Regardless of whether you opt to fix a row or column, the end result, which is the determinant of the matrix, will remain consistent.

   with j as a fixed value ranging from 1 to d,

或者，使用固定的i，

Or, with the fixed i,

根据以下等式，我们可以看到某些情况可以轻松计算：

Based on the following equations, we can see that some of the cases can be easily calculated:

• 对角矩阵：对于对角矩阵，行列式是其对角元素的乘积。
• Diagonal matrix: For a diagonal matrix, the determinant is the product of its diagonal elements.
• 三角矩阵：在上下文中对于三角矩阵，行列式是通过将其所有对角线元素相乘来找到的。如果矩阵的行或列的所有分量均为 0，则行列式也为 0。

  对于2 × 2 矩阵

  

  其行列式可以计算为ad - bc。如果我们考虑一个3 × 3矩阵，

  

  行列式计算如下：

• Triangular matrix: In the context of a triangular matrix, the determinant is found by multiplying all its diagonal elements. If all components of a matrix’s row or column are 0, the determinant is also 0.

  For a 2 × 2 matrix of

  Its determinant can be computed as ad - bc. If we consider a 3 × 3 matrix,

  The determinant is calculated as follows:

现在让我们继续特征值和向量。

Let’s now move on to eigenvalues and vectors.

特征值和特征向量

Eigenvalues and eigenvectors

属于d × d矩阵A 的向量x是特征向量，如果它满足方程Ax = λx，其中λ表示与矩阵相关的特征值。这种关系描绘了矩阵A与其对应的特征向量x之间的联系，可以看出作为矩阵的“拉伸方向”。如果A是可对角化的矩阵，则可以将其解构为d × d可逆矩阵V和对角d × d矩阵Δ，使得

A vector x, belonging to a d × d matrix A, is an eigenvector if it satisfies the equation Ax = λx, where λ represents the eigenvalue associated with the matrix. This relationship delineates the link between matrix A and its corresponding eigenvector x, which can be perceived as the “stretching direction” of the matrix. In the case where A is a matrix that can be diagonalized, it can be deconstructed into a d × d invertible matrix, V, and a diagonal d × d matrix, Δ, such that

V的列包含d特征向量，而Δ的对角线条目包含相应的特征值。线性变换Ax可以通过一系列的三个操作来直观地理解。最初， x的乘法计算与V的列相关的非正交基础上的 x坐标。随后， x 乘以Δ使用Δ中的因子缩放这些坐标，与特征向量的方向对齐。最后，与V相乘将坐标恢复到原始基础，从而导致沿d特征向量方向的各向异性缩放。

The columns of V encompass d eigenvectors, while the diagonal entries of Δ house the corresponding eigenvalues. The linear transformation Ax can be visually understood through a sequence of three operations. Initially, the multiplication of x by calculates x’s co-ordinates in a non-orthogonal basis associated with V’s columns. Subsequently, the multiplication of x by Δ scales these co-ordinates using the factors in Δ, aligned with the eigenvectors’ directions. Finally, the multiplication with V restores the co-ordinates to the original basis, resulting in an anisotropic scaling along the d eigenvector directions.

可对角矩阵表示涉及沿d线性独立方向的各向异性缩放的变换。当V的列是正交向量时，等于其转置 ，表示沿相互正交的方向缩放。在这种情况下，矩阵A始终可对角化，并且当V的列为正交向量，由以下关系确定。

Diagonalizable matrices signify transformations involving anisotropic scaling along d-linearly independent directions. When V ‘s columns are orthonormal vectors, equals its transpose, , indicating scaling along mutually orthogonal directions. In such cases, matrix A is always diagonalizable and exhibits symmetry when V’s columns are orthonormal vectors, as affirmed by the following relationship.

寻找特征向量的数值方法

Numerical methods for finding eigenvectors

确定d × d矩阵A的特征向量的传统方法涉及查找方程的d根：

The conventional method to ascertain the eigenvectors of a d × d matrix A involves locating the d roots, of the equation:

其中一些根可能是重复。后续步骤涉及求解 形式的线性系统，通常使用高斯消去法来实现。然而，这种方法可能并不总是最稳定或最精确的，因为多项式方程的求解器在实际应用中可能会表现出病态和数值不稳定。事实上，在工程中求解高次多项式方程的流行技术涉及构造具有与原始多项式相同的特征多项式的伴随矩阵，然后确定其特征值。

Some of these roots might be repeated. The subsequent step involves solving linear systems in the form , typically achieved using the Gaussian elimination method. However, this method might not always be the most stable or precise, as solvers of polynomial equations can exhibit ill-conditioning and numerical instability in practical applications. Indeed, a prevalent technique for resolving high-degree polynomial equations in engineering involves constructing a companion matrix possessing the same characteristic polynomial as the original polynomial and then determining its eigenvalues.

特征值分解

Eigenvalue decomposition

特征值分解，也称为矩阵的特征分解或对角化，是线性代数和计算数学中使用的强大数学工具。特征值分解的目标是将给定矩阵分解为表示矩阵的特征向量和特征值的矩阵的乘积。

Eigenvalue decomposition, also known as the eigen-decomposition or the diagonalization of a matrix, is a powerful mathematical tool used in linear algebra and computational mathematics. The goal of eigenvalue decomposition is to decompose a given matrix into a product of matrices that represent the eigenvectors and eigenvalues of the matrix.

矩阵A的特征值分解是将矩阵分解为两个矩阵的乘积：矩阵V和矩阵D。

The eigenvalue decomposition of matrix A is a factorization of the matrix into the product of two matrices: the matrix V and the matrix D.

V 的列是矩阵A的特征向量，D是对角矩阵，包含其诊断上相应的特征值。

V has column which are the eigenvectors of matrix A, and D is a diagonal matrix that contains the corresponding eigenvalues on its diagnol.

特征值问题是找到非 0 向量v和标量λ，使得Av = λv，其中A是方阵，因此v是A的特征向量。标量λ称为矩阵A的特征值。特征值问题可以写成矩阵形式为Av = λIv，其中I是单位矩阵。

The eigenvalue problem is to find the non-0 vectors, v, and the scalars, λ, such that Av = λv, where A is a square matrix, and thus v is an eigenvector of A. The scalar λ is called the eigenvalue of matrix A. The eigenvalue problem can be written in matrix form as Av = λIv, where I is the identity matrix.

确定的过程特征值与矩阵A的特征方程密切相关，该方程是从 导出的多项式方程。可以求解特征方程以获得特征值λ，即方程的根。一旦找到特征值，就可以通过求解线性方程组来找到特征向量。

The process of determining eigenvalues is intimately linked to the characteristic equation of matrix A, which is the polynomial equation derived from . The characteristic equation can be solved for the eigenvalues, λ, which are the roots of the equation. Once the eigenvalues are found, the eigenvectors can be found by solving the system of linear equations .

特征值分解的一个重要属性是它允许我们对矩阵进行对角化，这意味着我们可以通过使用适当的特征向量矩阵将矩阵转换为对角形式。矩阵的对角形式很有用，因为它允许我们轻松计算矩阵的迹和行列式。

One important property of eigenvalue decomposition is that it allows us to diagonalize a matrix, which means that we can transform the matrix into a diagonal form by using an appropriate eigenvectors matrix. The diagonal form of a matrix is useful because it allows us to calculate the trace and determinant of the matrix easily.

特征值分解的另一个重要属性是它提供了对矩阵结构的深入了解。例如，对称矩阵的特征值始终是实数，并且特征向量是正交的，这意味着它们彼此垂直。在非对称矩阵的情况下，特征值可以是复数，并且特征向量不一定是正交的。

Another important property of eigenvalue decomposition is that it provides insight into the structure of the matrix. For example, the eigenvalues of a symmetric matrix are always real, and the eigenvectors are orthogonal, which means that they are perpendicular to each other. In the case of non-symmetric matrices, the eigenvalues can be complex, and the eigenvectors are not necessarily orthogonal.

矩阵的特征值分解在数学、物理、工程和计算机科学中有许多应用。在数值分析中，特征值分解用于查找线性系统的解、计算矩阵的特征值以及查找矩阵的特征向量。在物理学中，特征值分解用于分析系统的稳定性，例如微分方程中平衡点的稳定性。在工程中，特征值分解用于研究系统的动力学，例如机械系统的振动。

The eigenvalue decomposition of a matrix has many applications in mathematics, physics, engineering, and computer science. In numerical analysis, eigenvalue decomposition is used to find the solution of linear systems, compute the eigenvalues of a matrix, and find the eigenvectors of a matrix. In physics, eigenvalue decomposition is used to analyze the stability of systems, such as the stability of an equilibrium point in a differential equation. In engineering, eigenvalue decomposition is used to study the dynamics of systems, such as the vibrations of a mechanical system.

在计算机科学领域，特征值分解在各个领域都有多种应用，包括机器学习和数据分析。在机器学习中，特征值分解在实现主成分分析( PCA )方面发挥着关键作用，这是一种技术 用于广泛数据集的降维。在数据分析领域，利用特征值分解来计算奇异值分解（SVD ），这是剖析和理解复杂情况的有力工具数据集。

Within the field of computer science, eigenvalue decomposition finds versatile applications across various domains, including machine learning and data analysis. In machine learning, eigenvalue decomposition plays a pivotal role in enabling principal component analysis (PCA), a technique employed for dimensionality reduction in extensive datasets. In the realm of data analysis, eigenvalue decomposition is harnessed to calculate the singular value decomposition (SVD), a potent tool for dissecting and understanding complex datasets.

奇异值分解

Singular value decomposition

最小化 的问题是一个典型的问题，其中x是具有单位范数的列向量，A是对称的d × d数据矩阵在许多机器学习环境中遇到的问题。这种问题类型经常出现在主成分分析、奇异值分解和谱聚类等应用中，所有这些都涉及特征工程和降维。优化问题可以表述如下：

The problem of minimizing , where x is a column vector that has a unit norm, and A is a symmetric d × d data matrix, is a typical problem encountered in numerous machine learning contexts. This problem type is often found in applications such as principal component analysis, singular value decomposition, and spectral clustering, all of which involve feature engineering and dimensionality reduction. The optimization problem can be articulated as follows:

最小化

Minimize

受

Subject to

我们可以将优化问题解决为最大化或最小化形式。施加向量x必须是单位向量的约束显着改变了优化问题的性质。与前一节相反，矩阵A的半正定性对于确定解不再至关重要。即使A不定，对范数的约束向量x确保了明确定义的解，防止涉及具有无界大小的向量或平凡的解，例如0向量。值奇异分解( SVD ) 是一种数学技术，它将矩形矩阵A分解为三个矩阵：U、S和。矩阵A定义为n × p矩阵。SVD 定理指出，A可以表示为三个矩阵的乘积：、其中、 和、U和V是正交矩阵。

We can solve the optimization problem as a maximization or minimization form. Imposing the constraint that vector x must be a unit vector significantly changes the nature of the optimization problem. In contrast to the prior section, the positive semi-definiteness of matrix A is no longer crucial for determining the solution. Even when A is indefinite, the constraint on the norm of vector x ensures a well-defined solution, preventing the involvement of vectors with unbounded magnitudes or trivial solutions, such as the 0 vector. Value singular decomposition (SVD) is a mathematical technique that takes a rectangular matrix, A, and decomposes it into three matrices: U, S, and . Matrix A is defined as an n × p matrix. The theorem of SVD states that A can be represented as the product of three matrices: , where , and , and U and V are orthogonal matrices.

U矩阵的列称为左奇异向量，而V矩阵转置的行称为右奇异向量。S矩阵具有奇异值，是与A大小相同的对角矩阵。SVD 将原始数据分解为坐标系，其中定义向量是正交的（正交和法线）。 SVD 计算涉及识别矩阵和的特征值和特征向量。矩阵V的列由 中的特征向量组成，矩阵U的列由 中的特征向量组成。S矩阵中的奇异值源自 或 的特征值的平方根，并按降序排列。这些奇异的值是实数。如果A是实数矩阵，则U和V也将是实数。

The U matrix’s columns are known as the left singular vectors, while the rows of the transpose of the V matrix are the right singular vectors. The S matrix, with singular values, is a diagonal matrix of the same size as A. SVD decomposes the original data into a co-ordinate system where the defining vectors are orthonormal (both orthogonal and normal). SVD computation involves identifying the eigenvalues and eigenvectors of matrices and . Matrix V’s columns consist of eigenvectors from , and matrix U’s columns consist of eigenvectors from . Singular values in the S matrix are derived from the square roots of eigenvalues from either or , organized in decreasing order. These singular values are real numbers. If A is a real matrix, U and V will also be real.

为了说明SVD的计算，提供一个例子。考虑一个4 × 2矩阵。可以通过计算然后确定这些矩阵的特征向量来找到矩阵的特征值。 U的列由 的特征向量形成，V的列由 的特征向量形成。S矩阵包含 或的特征值的平方根。通过求解给定示例中的特征方程即可找到特征值，其中W是矩阵，I是单位矩阵，λ是特征值。然后通过求解从特征值方程导出的方程组来找到特征向量。然后通过组合特征向量和奇异值获得最终矩阵U 、S、 和。

To illustrate the calculation of SVD, an example is provided. Consider a 4 × 2 matrix. The eigenvalues of the matrix can be found by computing and and then determining the eigenvectors of these matrices. U’s columns are formed by the eigenvectors of , and V’s columns are formed by the eigenvectors of . The S matrix comprises the square root of eigenvalues from either or . Eigenvalues are found by solving the characteristic equation in the given example , where W is the matrix, I is the unit matrix, and λ is the eigenvalue. The eigenvectors are then found by solving the set of equations derived from the eigenvalue equations. The final matrices U, S, and are then obtained by combining the eigenvectors and singular values.

需要注意的是，奇异值是按降序排列的，其中

It should be noted that the singular values are in descending order, with

现在让我们继续学习机器学习的基本概率。

Let’s now move on to basic probability for machine learning.

机器学习的基本概率

Basic probability for machine learning

概率提供信息关于事件发生的可能性。在该领域，有几个重要术语需要理解：

Probability provides information about the likelihood of an event occurring. In this field, there are several key terms that are important to understand:

• 试验或实验：以一定的可能性产生一定结果的行动
• Trial or experiment: An action that results in a certain outcome with a certain likelihood
• 样本空间：这包含给定实验的所有潜在结果
• Sample space: This encompasses all potential outcomes of a given experiment
• 事件：这表示样本空间的非空部分
• Event: This denotes a non-empty portion of the sample space

因此，用技术术语来说，概率是对进行实验时事件发生的可能性的度量。

Therefore, in technical terms, probability is a measure of the likelihood of an event occurring when an experiment is conducted.

在这个非常简单的例子中，事件A产生一个结果的概率等于事件A发生的概率除以所有可能发生的事件的概率。例如，在掷一枚均匀的硬币时，有相同机会出现两种结果：正面和反面。出现正面的概率为1/(1+1) = ½。

In this very simple case, the probability of event A with one outcome is equal to the chance of event A divided by the chance of all possible events. For example, in flipping a fair coin, there are two outcomes with the same chance: heads and tails. The chance of having heads will be 1/(1+1) = ½.

为了计算概率，给定一个具有n 个结果的事件A和一个样本空间S，事件 A 的概率计算如下

In order to calculate the probability, given an event, A, with n outcomes and a sample space, S, the probability of event A is calculated as

其中代表A中的结果。假设实验的所有结果具有相同的概率，并且其中一个结果的选择不会影响后续轮次中其他结果的选择（意味着它们在统计上是独立的），那么

where represents the outcomes in A. Assuming all results of the experiment have equal probability, and the selection of one does not influence the selection of others in subsequent rounds (meaning they are statistically independent), then

因此，概率值的范围是0到1，样本空间体现了完整的潜在结果集，表示为P(S) = 1。

Hence, the value of probability ranges from 0 to 1, with the sample space embodying the complete set of potential outcomes, denoted as P(S) = 1.

统计上独立

Statistically independent

在统计领域，有两个如果一个事件的发生不影响另一事件发生的可能性，则事件被定义为独立的。正式地说，当P(A 和 B) = P(A)P(B)时，事件A和B是独立的，其中P(A)和P(B)分别是事件A和B发生的概率。

In the realm of statistics, two events are defined as independent if the occurrence of one event doesn’t influence the likelihood of the other event’s occurrence. To put it formally, events A and B are independent precisely when P(A and B) = P(A)P(B), where P(A) and P(B) are the respective probabilities of events A and B happening.

考虑这个例子来阐明统计独立性的概念：假设我们拥有两枚硬币，一枚是公平的（出现正面或反面的机会均等），另一枚是有偏差的（显示正面比反面更有可能）。如果我们翻转公平的硬币和有偏差的硬币，这两个事件在统计上是独立的，因为一枚硬币翻转的结果不会改变概率另一枚硬币正面或反面朝上。具体来说，两枚硬币正面朝上的可能性是各个概率的乘积：(1/2) * (3/4) = 3/8。

Consider this example to clarify the concept of statistical independence: imagine we possess two coins, one fair (an equal chance of turning up heads or tails) and the other biased (showing a head is more likely than a tail). If we flip the fair coin and the biased coin, these two events are statistically independent because the outcome of one coin flip doesn’t alter the probability of the other coin turning up heads or tails. Specifically, the likelihood of both coins showing heads is the product of the individual probabilities: (1/2) * (3/4) = 3/8.

统计独立性是统计学和概率论中的一个关键概念，在机器学习中经常被用来概述数据集中变量之间的联系。通过理解这些关系，机器学习算法可以更好地发现模式并提供更精确的预测。我们将在下面描述不同类型事件之间的关系：

Statistical independence is a pivotal concept in statistics and probability theory, frequently leveraged in machine learning to outline the connections between variables within a dataset. By comprehending these relationships, machine learning algorithms can better spot patterns and deliver more precise predictions. We will describe the relationship between different types of events in the following:

• 互补事件： A的互补事件，表示为A'，包含所有事件的概率样本空间中未包含在 A 中的潜在结果。了解A和A'在统计上独立是至关重要的：
• Complementary event: The complementary event to A, signified as A’, encompasses the probability of all potential outcomes in the sample space not included in A. It’s critical to understand that A and A’ are statistically independent:

• 并集和交集： A的互补事件，表示为A'，包含样本空间中所有潜在结果不存在的概率。包含在A中。了解A和A'在统计上独立是至关重要的。
• Union and intersection: The complementary event to A, signified as A’, encompasses the probability of all potential outcomes in the sample space not included in A. It’s critical to understand that A and A’ are statistically independent.
• 互斥：当两个事件没有共同的结果，它们被视为相互排斥的。换句话说，如果A和B是互斥事件，则P(A ∩ B) = 0。这个结论可以从概率的加法规则得出，如A和B a重新脱节的事件：
• Mutually exclusive: When two events have no shared outcomes, they are viewed as mutually exclusive. In other words, if A and B are mutually exclusive events, then P(A ∩ B) = 0. This conclusion can be drawn from the addition rule of probability, as A and B are disjointed events:

• 独立：当一个事件的发生不影响事件的发生时，两个事件被认为是独立的对方的发生。如果A和B是两个独立事件，那么
• Independent: Two events are deemed independent when the occurrence of one doesn’t impact the occurrence of the other. If A and B are two independent events, then

接下来，我们将描述离散随机变量、它的分布以及如何使用它来计算概率。

Next, we are going to describe the discrete random variable, its distribution, and how to use it to calculate the probabilities.

离散随机变量及其分布

Discrete random variables and their distribution

离散随机变量指的是可以假设有限或可数无限数量的潜在结果的变量。此类变量的示例可能是抛硬币产生的正面计数、特定时间范围内通过收费站的汽车计数或教室中金发学生的数量。